Enhanced clear path detection in the presence of traffic infrastructure indicator

ABSTRACT

A method for detecting a clear path of travel for a vehicle utilizing analysis of a plurality of images generated by a camera device located upon the vehicle includes monitoring the images. The images are analyzed including determining a clear path upon which a potential road surface can be estimated from other portions of the images that do not indicate a potential road surface, and determining an image of a traffic infrastructure indication. The method further includes determining the content of the traffic infrastructure indication, modifying the clear path based upon the content of the traffic infrastructure indication, and utilizing the modified clear path in navigation of the vehicle.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of U.S. application Ser.No. 12/581,849 filed on Oct. 19, 2009, which is a continuation-in-partof U.S. application Ser. No. 12/581,659 filed on Oct. 19, 2009, which isa continuation-in-part of U.S. application Ser. No. 12/474,594 filed onMay 29, 2009, which is a continuation-in-part of U.S. application Ser.No. 12/108,581 filed on Apr. 24, 2008. U.S. application Ser. No.12/581,659 filed on Oct. 19, 2009 claims the benefit of U.S. ProvisionalApplication 61/215,745 filed on May 8, 2009. U.S. application Ser. No.12/581,849, U.S. application Ser. No. 12/581,659, U.S. application Ser.No. 12/474,594, U.S. application Ser. No. 12/108,581, and U.S.Provisional Application 61/215,745 are incorporated herein by reference.

TECHNICAL FIELD

This disclosure is related to automated or semi-automated control of amotor vehicle.

BACKGROUND

The statements in this section merely provide background informationrelated to the present disclosure and may not constitute prior art.

Autonomous driving systems and semi-autonomous driving systems utilizeinputs regarding the road and other driving conditions to automaticallycontrol throttle and steering mechanisms. Accurate estimation andidentification of a clear path over which to desirably operate a motorvehicle is critical to successfully replacing the human mind as acontrol mechanism for vehicle operation.

Road conditions can be complex. Under normal operation of a vehicle, thehuman operator makes hundreds of observations per minute and adjustsoperation of the vehicle on the basis of perceived road conditions. Oneaspect of perceiving road conditions is the perception of the road inthe context of objects in and around the roadway and navigating a clearpath through any objects. Replacing human perception with technologypreferentially includes some means to accurately perceive objects andcontinue to effectively navigate around such objects.

Technological means for perceiving an object or road conditions includedata from visual cameras, radar imaging, LIDAR, vehicle to vehiclecommunications, vehicle to infrastructure communications, and use ofglobal positioning data with a digital map. Cameras translate visualimages in the form of radiation such as light patterns or infraredsignatures into a readable data format. One such data format includespixelated images, in which a perceived scene is broken down into aseries of pixels. Radar imaging utilizes radio waves generated by atransmitter to estimate shapes and objects present in front of thetransmitter. Patterns in the waves reflecting off these shapes andobjects can be analyzed and the locations of objects can be estimated.

Once data has been generated regarding the ground in front of thevehicle, the data must be analyzed to estimate the presence of objectsfrom the data. By using cameras and radar imaging systems, ground orroadway in front of the vehicle can be analyzed for the presence ofobjects that might need to be avoided. However, the mere identificationof potential objects to be avoided does not complete the analysis. Animportant component of any autonomous system includes how potentialobjects identified in perceived ground data are processed andmanipulated to identify a clear path in which to operate the vehicle.

One known method to identify a clear path in which to operate thevehicle is to catalog and provisionally identify all perceived objectsand identify a clear path in light of the locations and behaviors ofidentified objects. Images may be processed to identify and classifyobjects according to their form and relationship to the roadway. Whilethis method can be effective in identifying a clear path, it requires agreat deal of processing power, requiring the recognition and separationof different objects in the visual image, for instance, distinguishingbetween a tree along the side of the road and a pedestrian walkingtoward the curb. Such methods can be slow or ineffective to processcomplex situations or may require bulky and expensive equipment tosupply the necessary processing capacity.

SUMMARY

A method for detecting a clear path of travel for a vehicle utilizinganalysis of a plurality of images generated by a camera device locatedupon the vehicle includes monitoring the images. The images are analyzedincluding determining a clear path upon which a potential road surfacecan be estimated from other portions of the images that do not indicatea potential road surface, and determining an image of a trafficinfrastructure indication. The method further includes determining thecontent of the traffic infrastructure indication, modifying the clearpath based upon the content of the traffic infrastructure indication,and utilizing the modified clear path in navigation of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments will now be described, by way of example, withreference to the accompanying drawings, in which:

FIG. 1 illustrates an exemplary arrangement of a vehicle equipped with acamera and a radar imaging system in accordance with the disclosure;

FIG. 2 illustrates a known method to determine a clear path forautonomous driving in accordance with the disclosure;

FIG. 3 illustrates an exemplary method to determine a clear pathutilizing a likelihood analysis of an image in accordance with thedisclosure;

FIG. 4 illustrates an exemplary method to analyze an image in accordancewith the disclosure;

FIG. 5 illustrates an exemplary method to define a classification errorby tuning a single threshold in accordance with the disclosure;

FIGS. 6A, 6B, and 6C illustrate an exemplary determination of an imagedifference by calculating an absolute image intensity difference inaccordance with the disclosure;

FIG. 7 illustrates an exemplary method to classify a feature as aportion of a clear path and as a detected object at the same time as amethod of image analysis in accordance with the disclosure;

FIG. 8 further illustrates an exemplary method to classify a feature asa portion of a clear path and as a detected object at the same time as amethod of image analysis in accordance with the disclosure;

FIG. 9 illustrates an exemplary process to analyze an image throughlikelihood analysis in accordance with the disclosure;

FIG. 10 is a first processing scheme for detecting a clear path, inaccordance with the disclosure;

FIG. 11 shows an exemplary current image including an exemplary set ofinterest points identified on the current image, in accordance with thepresent disclosure;

FIG. 12 shows an exemplary current image captured subsequent to theimage depicted in FIG. 11, in accordance with the present disclosure;

FIGS. 13A and 13B show an exemplary matched point pair, the matchedpoint pair comprising an interest point from a current image and acorresponding interest point from a previous image, and an exemplarypixel region surrounding the points, in accordance with the presentdisclosure;

FIG. 13A depicts the interest point from the current image and the pixelregion surround the interest point;

FIG. 13B depicts the interest point from the previous image and thepixel region surround the interest point;

FIG. 14 graphically illustrates use of an exemplary current image, anexemplary previous image, and operation of a host vehicle to determinethe position of a viewed feature, in accordance with the presentdisclosure;

FIG. 15 graphically illustrates an exemplary overhead map describingvertical positions of interest points at x and y coordinates in front ofthe host vehicle, in accordance with the present disclosure;

FIG. 16 shows a second processing scheme for detecting a clear path, inaccordance with the present disclosure;

FIG. 17 shows an exemplary filtered image based upon pixel colorintensity, in accordance with the present disclosure;

FIG. 18 shows an exemplary filtered image utilizing an edge recognitionmethod, in accordance with the present disclosure;

FIG. 19 depicts an alternate exemplary process for the second processingscheme, in accordance with the present disclosure;

FIG. 20 depicts an exemplary flow chart for detecting an enhanced clearpath, in accordance with the present disclosure;

FIG. 21 depicts an exemplary flow chart for detecting an enhanced clearpath, in accordance with the present disclosure; and

FIG. 22 graphically depicts such an iterative analysis, in accordancewith the present disclosure.

FIG. 23 depicts an exemplary flow chart for detecting an enhanced clearpath, in accordance with the present disclosure.

DETAILED DESCRIPTION

Referring now to the drawings, wherein the showings are for the purposeof illustrating certain exemplary embodiments only and not for thepurpose of limiting the same, FIG. 1 illustrates an exemplaryarrangement of camera 110 located on the front of a vehicle 100 andpointed toward the ground in front of the vehicle 100 in accordance withthe disclosure. Camera 110 is in communication with processing module120 containing logic to process inputs from camera 110. The vehicle 100may also be equipped with a radar imaging system 130, which, whenpresent, is also in communication with processing module 120. It shouldbe appreciated by those having ordinary skill in the art that thevehicle 100 could utilize a number of methods to identify roadconditions in addition or in the alternative to the use of camera 110and the radar imaging system 130, including GPS information, informationfrom other vehicles in communication with the vehicle 100, historicaldata concerning the particular roadway, biometric information such assystems reading the visual focus of the driver, a radar imaging system,or other similar systems. The disclosure herein can be applied tovarious device arrangements and is therefore not limited thereby.

The camera 110 is a device well known in the art capable of translatingvisual inputs in the form of light, infrared, or other electro-magnetic(EM) radiation into a data format readily capable of analysis, e.g., adigital, pixelated image. In one embodiment, the camera 110 uses acharge coupled device (CCD) sensor to generate images indicating afield-of-view. Preferably, the camera 110 is configured for continuousimage generation, e.g., 30 images generated per second. Images generatedby the camera 110 may be stored in memory within the camera 110 ortransferred to the processing module 120 for storage and/or analysis.Preferably, each image generated by the camera 110 is a two-dimensionalimage of known pixel dimensions comprising a plurality of identifiablepixels. The plurality of identifiable pixels may be stored and analyzedusing an array. Each pixel may be represented in the array as a set ofbits or a plurality of sets of bits wherein the bits correspond to acolor on a predetermined palette or color map. Each pixel may beexpressed as a function of a plurality of color intensity values such asin a red-green-blue (RGB) color model or a cyan-magenta-yellow-key(CMYK) color model. Preferably, each pixel comprises a plurality of setsof bits wherein each set of bits corresponds to a color intensity and acolor intensity value e.g., a first set of bits corresponds to a redcolor intensity value, a second set of bits corresponds to a green colorintensity value, and a third set of bits corresponds to blue colorintensity value on the RGB color model.

The radar imaging device 130 is a device well known in the artincorporating a transmitter capable of emitting radio waves or other EMradiation, a receiver device capable of sensing the emitted wavesreflected back to the receiver from objects in front of the transmitter,and means to transfer the sensed waves into a data format capable ofanalysis, indicating for example range and angle from the objects offwhich the waves reflected. Alternatively, the radar imaging device 130may be replaced or supplemented with a light detection and ranging(LIDAR) system configured to transmit and receive optical energy. Thereceived optical energy may be used to determine object geometricdimensions and/or geometrical proximity to the vehicle 100. It will benoted that radar imaging device 130 is optional and unnecessary toperform many of the methods disclosed herein, wherein processing ofvisual images is capable of accomplishing clear path detection.

The processing module 120 is illustrated in FIG. 1, and described hereinas a discrete element. Such illustration is for ease of description andit should be recognized that the functions performed by this element maybe combined in one or more devices, e.g., implemented in software,hardware, and/or application-specific integrated circuitry. Theprocessing module 120 can be a general-purpose digital computercomprising a microprocessor or central processing unit, storage mediumscomprising non-volatile memory including read only memory andelectrically programmable read only memory, random access memory, a highspeed clock, analog to digital and digital to analog circuitry, andinput/output circuitry and devices and appropriate signal conditioningand buffer circuitry. In the alternative, processing module 120 can be adigital signal processing (DSP) unit, such as a customized integratedcircuit such as a field programmable gate array. The processing module120 has a set of processing algorithms, comprising resident programinstructions and calibrations stored in the non-volatile memory andexecuted to provide desired functions. The algorithms are preferablyexecuted during preset loop cycles. Algorithms are executed by thecentral processing unit and are operable to monitor inputs from theaforementioned sensing devices and execute control and diagnosticroutines to control operation of the actuators, using presetcalibrations. Loop cycles may be executed at regular intervals, forexample each 3.125, 6.25, 12.5, 25 and 100 milliseconds during ongoingengine and vehicle operation. Alternatively, algorithms may be executedin response to occurrence of an event.

The processing module 120 executes algorithmic code stored therein tomonitor related equipment such as camera 110 and radar imaging system130 and execute commands or data transfers as indicated by analysisperformed within the processing module. Processing module 120 mayinclude algorithms and mechanisms to actuate autonomous driving controlby means known in the art and not described herein, or processing module120 may simply provide information to a separate autonomous drivingsystem. Processing module 120 is adapted to receive input signals fromother systems and the operator as necessary depending upon the exactembodiment utilized in conjunction with the control module.

FIG. 2 illustrates a known method to determine a clear path forautonomous driving in accordance with the disclosure. Image 10 isgenerated corresponding to the roadway in front of vehicle 100. Throughone of various methods, objects 40A, 40B, and 40C are identified withinimage 10, and each object is categorized and classified according tofiltering and trained object behaviors. Separate treatment of eachobject can be computationally intensive, and requires expensive andbulky equipment to handle the computational load. An algorithm processesall available information regarding the roadway and objects 40 toestimate a clear path available to vehicle 100. Determination of theclear path depends upon the particular classifications and behaviors ofthe identified objects 40.

FIG. 3 illustrates an exemplary method to determine a clear path forautonomous or semi-autonomous driving in accordance with the disclosure.Image 10 is depicted including ground 20, horizon 30, and objects 40.Image 10 is collected by camera 110 and represents the road environmentin front of vehicle 100. Ground 20 represents the zone of all availablepaths open to travel without any discrimination on the basis of objectsthat might be present. The method of FIG. 3 that determines a clear pathupon ground 20 starts by presuming all of ground 20 is clear, and thenutilizes available data to disqualify portions of ground 20 as notclear. In contrast to the method of FIG. 2 which classifies every object40, the method of FIG. 3 instead analyzes ground 20 and seeks to definea likelihood from available data that some detectable anomaly which mayrepresent object 40 limits or makes not clear that portion of ground 20.This focus upon ground 20 instead of objects 40 avoids the complexcomputational tasks associated with managing the detection of theobjects. Individual classification and tracking of individual objects isunnecessary, as individual objects 40 are simply grouped together as apart of the overall uniform limitation upon ground 20. Ground 20,described above as all paths open to travel without discrimination,minus limits placed on ground 20 by areas found to be not clear, defineclear path 50, depicted in FIG. 3 as the area within the dotted lines,or an area with some threshold likelihood of being open for travel ofvehicle 100.

Object 40 that creates not clear limitations upon ground 20 can takemany forms. For example, an object 40 can represent a discrete objectsuch as a parked car, a pedestrian, or a road obstacle, or object 40 canalso represent a less discreet change to surface patterns indicating anedge to a road, such as a road-side curb, a grass line, or watercovering the roadway. Object 40 can also include an absence of flat roadassociated with ground 20, for instance, as might be detected with alarge hole in the road. Object 40 can also include an indicator withoutany definable change in height from the road, but with distinct clearpath implications for that segment of road, such as a paint pattern onthe roadway indicative of a lane marker. The method disclosed herein, bynot seeking to identify object 40 but merely to take visual cues fromground 20 and anything in proximity to the ground in image 10, evaluatesa likelihood of clear versus not clear and adjusts the control ofvehicle 100 for the presence of any object 40.

Numerous methods for automated analysis of two-dimensional (2D) imagesare possible. Analysis of image 10 is performed by an algorithm withinprocessing module 120. FIG. 4 illustrates one exemplary method which maybe applied to analyze image 10 in accordance with the disclosure. Thismethod subdivides image 10 and identifies a sub-image or patch 60 ofground 20 for analysis, extracts features or analyzes the availablevisual information from patch 60 to identify any interesting ordistinguishing features within the patch, and classifies the patchaccording to a likelihood of being a clear path according to analysis ofthe features. Patches with greater than a certain threshold oflikeliness are classified as clear, and a compilation of patches can beused to assemble a clear path within the image.

Patch 60, as a sub-image of image 10, can be identified through anyknown means, such as random search or swarm search of image 10.Alternatively, information regarding the presence of an object 40available from some other source of information, such as radar imagingsystem 130, can be used to identify a patch to analyze the portion ofimage 10 which should describe object 40. Image 10 may require manypatches 60 to analyze the whole image. In addition, multiple overlayingpatches or patches of different size could be used to fully analyze aregion of image 10 containing information of interest. For instance, asmall patch 60 might be used to analyze a small dot on the road;however, a large patch 60 might be required to analyze a series of dotswhich in isolation might seem uninteresting, but in context of theentire series, could indicate an object 40 of interest. In addition, theresolution of patches applied to a particular area may be modulatedbased upon information available, for instance, with more patches beingapplied to a region of image 10 wherein an object 40 is thought toexist. Many schemes or strategies can be utilized to define patches 60for analysis, and the disclosure is not intended to be limited to thespecific embodiments described herein.

Once a patch 60 has been identified for analysis, processing module 120processes the patch by application of a filter to extract features fromthe patch. Additionally, processing module 120 may perform analysis ofthe location of the patch in context to the location of the vehicle.Filters utilized may take many forms. Filtering algorithms utilized toextract features often search the available visual information forcharacteristic patterns in the data, defining features by lineorientation, line location, color, corner characteristics, other visualattributes, and learned attributes. Learned attributes may be learned bymachine learning algorithms within the vehicle, but are most frequentlyprogrammed offline and may be developed experimentally, empirically,predictively, through modeling or other techniques adequate toaccurately train distinguishing attributes.

Once features in patch 60 have been extracted, the patch is classifiedon the basis of the features to determine the likelihood that the patchis a clear path. Likelihood analysis is a process known in the art bywhich a likelihood value or a confidence is developed that a particularcondition exists. Applied to the present disclosure, classificationincludes likelihood analysis to determine whether the patch represents aclear path or if ground 20 in this patch is limited by an object 40.Classification is performed in an exemplary embodiment by application ofclassifiers or algorithms trained with a database of exemplary roadconditions and interactions with detected objects. These classifiersallow processing module 120 to develop a fractional clear pathlikelihood value for patch 60, quantifying a confidence between zero andone that the features identified within the patch do not indicate alimiting object 40 which would inhibit free travel of vehicle 100. Athreshold confidence can be set, defining the clear path likelihoodrequired to define the patch as a clear path, for instance by thefollowing logic:Confidence=ClearPathLikelihood(i)If _Confidence>0.5, then _patch=clearpath  (1)In this particular exemplary embodiment, a confidence of 50% or 0.5 isselected as the threshold confidence. This number can be developedexperimentally, empirically, predictively, through modeling or othertechniques adequate to accurately evaluate patches for clear pathcharacteristics.

The likelihood analysis, as mentioned above, may be performed in oneexemplary embodiment by application of trained classifiers to featuresextracted from a patch. One method analyzes the features a-priori usinga training set of images. In this training stage, distinguishingfeatures are selected from a raw feature set, the distinguishingfeatures being defined by methods known in the art, such as Haarwavelet, Gabor wavelet, and Leung-and-Malik filter bank. In addition, 2Dimage location information based on each feature's minimalclassification errors, calculated as the sum of false acceptance rate(FAR) and false rejection rate (FRR), may be utilized by tuning a singlethreshold as illustrated in FIG. 5. This classification error can bedescribed through the following expression:ClassificctionError(i)=FAR_(i)+FRR_(i)  (2)Information from the trained classifiers is used to classify or weightthe feature as indicating a clear path or not clear path, the particularclassification depending upon the strength of comparisons to the traineddata. Classification of the feature, if the feature is the only featurewithin the patch, may be directly applied to the patch. Classificationof a patch with multiple features identified may take many forms,including the patch being defined by the included feature mostindicative of the patch being not clear or the patch being defined by aweighted sum of all of the features included therein.

The above method can be utilized to examine an individual image 10 andestimate a clear path 50 based upon visual information contained withinimage 10. This method may be repeated at some interval as the vehicletravels down the road to take new information into account and extendthe formulated clear path to some range in front of the vehicle's newposition. Selection of the interval must update image 10 with enoughfrequency to accurately supply vehicle 100 with a clear path in which todrive. However, the interval can also be selected to some minimum valueto adequately control the vehicle but also not to unduly burden thecomputational load placed upon processing module 120.

Clear path detection can be accomplished through a single image 10 asdescribed above. However, processing speed and accuracy can be improvedwith the addition of a second image taken in close time proximity to theoriginal image, such as sequential images from a streaming video clip. Asecond image allows direct comparison to the first and provides forupdated information regarding progression of the vehicle and movement ofdetected objects. Also, the change of perspective of camera 110 allowsfor different analysis of features from the first image: a feature thatmay not have shown up clearly or was indistinct in the first image maydisplay at a different camera angle, stand out more distinctly, or mayhave moved since the first image, allowing the classification algorithman additional opportunity to define the feature.

Processing of a second image in relation to the original image 10 can beperformed by calculating an image difference. If the image difference ofa point of interest, such as a feature identified by radar, is not zero,then the point can be identified as embodying new information. Pointswhere the image difference does equal zero can be eliminated fromanalysis and computation resources may be conserved. Methods todetermine image difference include absolute image intensity differenceand vehicle-motion compensated image difference.

Determining an image difference by calculating an absolute imageintensity difference can be used to gather information between twoimages. One method of absolute image intensity difference includesdetermining equivalent image characteristics between the original imageand the second image in order to compensate for movement in the vehiclebetween the images, overlaying the images, and noting any significantchange in intensity between the images. A comparison between the imagesindicating a change in image intensity in a certain area contains newinformation. Areas or patches displaying no change in intensity can bede-emphasized in analysis, whereas areas displaying clear changes inintensity can be focused upon, utilizing aforementioned methods toanalyze patches on either or both captured images.

FIGS. 6A, 6B, and 6C illustrate an exemplary determination of an imagedifference by calculating an absolute image intensity difference inaccordance with the disclosure. FIG. 6A depicts an original image. FIG.6B depicts a second image with changes from the original image. Inparticular the depicted circular shape has shifted to the left. Acomparison of the two images as illustrated in FIG. 6C, an outputrepresenting the result of an absolute image intensity differencecomparison, identifies one region having gotten darker from the firstimage to the second image and another region having gotten lighter fromthe first image to the second image. Such a method can be described asdifferencing. Analysis of the comparison yields information that somechange as a result of movement or change of perspective is likelyavailable in that region of the images. In this way, absolute imageintensity difference can be used to analyze a pair of sequential imagesto identify a potentially not clear path.

Likewise, determining an image difference by calculating avehicle-motion compensated image difference can be used to gatherinformation between two images. Many methods to calculate avehicle-motion compensated image difference are known. One exemplarymethod of vehicle-motion compensated image difference includes analyzinga potential object as both a stationary portion of a clear path and adetected object at the same time. Likelihood analysis is performed onfeatures identified corresponding to the potential object from bothclassifications at the same time, and the classifications may becompared, for example, through the following logic:Confidence(i)=ClearPathLikelihood(i)−DetectedObjectLikelihood(i)If _Confidence>0, then _patch=clearpath  (3)In this exemplary comparison, if confidence(i) is greater than zero,then the patch containing the feature is classified as a clear path. Ifconfidence(i) equals or is less than zero, then the patch containing thefeature is classified as not a clear path or limited. However, differentvalues may be selected for the confidence level to classify the patch asa clear path. For example, testing may show that false positives aremore likely than false negatives, so some factor or offset can beintroduced.

FIG. 7 illustrates one method to classify a feature as a portion of aclear path and as a detected object at the same time as described abovein accordance with the disclosure. Image 10 includes object 40,trapezoidal projection 70, and rectangular projection 80. This methodutilizes an assumption projecting object 40 as a flat object on theground within projection 70 to test the classification of the feature asa portion of a clear path. The method also utilized an assumptionprojecting object 40 as a vertical object within rectangular projection80 to test the classification of the feature as a detected object. FIG.8 illustrates comparisons made in data collected between the two imagesto evaluate the nature of object 40 in accordance with the disclosure.Camera 110 at time t₁ observes and captures data from object 40 in theform of a first image. If object 40 is an actual detected object, theprofile observed by camera 110 of object 40 at time t₁ will correspondto point 90A. If object 40 is a flat object in the same plane as ground20, then the profile observed by camera 110 of object 40 at time t₁ willcorrespond to point 90B. Between times t₁ and t₂, camera 110 travelssome distance. A second image is captured at time t2, and informationregarding object 40 can be tested by applying an algorithm looking atvisible attributes of the object in the second image in comparison tothe first image. If object 40 is an actual detected object, extendingupward from ground 20, then the profile of object 40 at time t₂ will beobserved at point 90C. If object 40 is a flat object in the same planeas ground 20, then the profile of object 40 at time t2 will be observedat point 90B. The comparison derived through vehicle-motion compensatedimage difference can directly assign a confidence by application ofclassifiers based on the observations of points 90, or the comparisonmay simply point to the area displaying change as a point of interest.Testing of the object against both classifications, as a flat object andas an actual detected object, allows either the area including object 40to be identified for further analysis through analysis of a patch asdescribed above or direct development of a clear path likelihood and adetected object likelihood for comparison, for example as in logicexpression (3) above.

Information available from analysis of the second image can additionallybe improved by integration of information regarding movement of thevehicle, such as speed and yaw-rate. Information regarding vehiclemotion is available from a number of sources, including the vehiclespeedometer, vehicle dynamic sensors or wheel speed sensors, anti-lockbraking mechanisms, and GPS location systems. Algorithms may utilizethis vehicle movement information, for example, in conjunction with theprojections described in FIGS. 7 and 8 to project angles which shouldexist in a feature laying flat on the ground in the second image basedupon data from the first image and the measured movement of the vehiclebetween the images.

The number of images utilized for comparison need not be limited to two.Multiple image analysis can be performed at multiple iterations, with anobject being tracked and compared over a number of cycles. As mentionedabove, computational efficiency can be gained by utilizing imagedifference analysis to identify points of interest and eliminating areaswith zero difference from subsequent analyses. Such efficiencies can beused in multiple iterations, for example, saying that only points ofinterest identified between a first and second image will be analyzed inthe third and fourth images taken. At some point, a fresh set of imageswill need to be compared to ensure that none of the areas showing zerodifference have had any change, for example a moving object impingingupon a previously clear path. The utilization of image differenceanalyses and of focused analyses, eliminating areas identified with zerochange, will vary from application to application and may vary betweendifferent operating conditions, such as vehicle speed or perceivedoperating environment. The particular utilization of image differenceanalyses and of focused analyses can take many different embodiments,and the disclosure is not intended to be limited to the specificembodiments described herein.

FIG. 9 illustrates an exemplary process 200 wherein input from a camerais analyzed to determine a clear path likelihood in accordance with thedisclosure. Camera input in the form of an image is generated at step202. At step 204, a patch is selected for analysis from the image. Step206 represents a filter or set of filters available to process thepatch. At step 208, feature extraction is performed upon the selectedpatch through application of filters available from step 206 andapplication of other algorithms. It will be appreciated that somefeatures have meaning and others do not, and a process of featureselection can be utilized to identify a group of best features foranalysis. Step 210 includes a classifier training process. As mentionedabove, classifiers or logic used in developing likelihood values areinitially trained offline. Training may optionally be continued in thevehicle based upon fuzzy logic, neural networks, or other learningmechanisms known in the art. These trained classifiers are utilized instep 212 to perform a likelihood analysis upon the features extracted instep 208, and a likelihood value for the patch is developed. Thislikelihood value expresses a confidence that the selected patch isclear. At step 214, the likelihood value developed in step 212 iscompared to a threshold likelihood value. If the likelihood value isgreater than the threshold value, then at step 218 the patch isidentified as a clear path. If the likelihood value is not greater thanthe threshold value, then the patch is identified as a not clear path.As described above, process 200 may be repeated or reiterated in anumber of ways, with the same image being analyzed repeatedly with theselection and analysis of different patches, and an identified patch canbe tracked and analyzed for change over a number of sequential images.

As mentioned above, processing module 120 may include algorithms andmechanisms to actuate autonomous driving control by means known in theart and not described herein, or processing module 120 may simplyprovide information to a separate autonomous driving system. Reactionsto perceived objects can vary, and include but are not limited tosteering changes, throttle changes, braking responses, and warning andrelinquishing control of the vehicle to the operator.

The above method, utilizing analysis of patches, is one method toestablish a clear path of travel for a vehicle. Two additional, relatedprocessing schemes are disclosed, employing similar methods to analyzepixels instead of patches. A first processing scheme utilizingtexture-rich methods to analyze images to identify a clear path isdisclosed. FIG. 10 shows a first processing scheme 101 for detecting aclear path, described herein as comprising discrete elements. Suchillustration is for ease of description and it should be recognized thatthe functions performed by these elements may be combined in one or moredevices, e.g., implemented in software, hardware, and/orapplication-specific integrated circuitry.

Processing scheme 101 is an exemplary method to apply texture-rich imageanalysis of a field-of-view in front of the host vehicle and begins withstep 103, wherein images of the field-of-view in front of the vehicleare generated. At step 106, the monitored images are examined forinterest points, for example, examining pixel color intensity asdescribed hereinabove and comparing the pixel or a group of pixels withsurrounding pixels. Through methods known in the art, interest pointscan be identified and utilized to accomplish methods described herein.At step 109, sequential images of the field-of-view in front of thevehicle are compared when the vehicle is in motion, and interest pointsfrom each image are matched to corresponding points in sequential imageswhich correspond to the same points in the field-of-view, wherepossible. Matching includes locating corresponding points throughtemplate matching or comparing interest points on the sequential images,taking into account movement of the host vehicle, and making a bestestimate whether two points represent the same object or feature visiblein the field-of-view. While interest points can be matched, not allmatched corresponding point pairs represent high quality correspondingpoint pairs that allow the identification of their three-dimensionalpositions in the field-of-view for classifications as a clear path forthe vehicle to travel through. At step 112, a filter is applied to thematched corresponding point pairs in order to identify high qualitycorresponding point pairs that can be utilized for three-dimensionalposition identification with high confidence. At step 115, the highquality corresponding point pairs are analyzed to determinethree-dimensional positions of objects represented by the correspondingpoint pairs. It will be appreciated that corresponding points atdifferent heights as compared to ground level will move differentlybetween sequential images. Analyzing movement of the interest points canyield estimated three dimensional coordinates of the interest points. Atstep 118, the determined object positions are utilized to map objectpositions in front of the host vehicle. At step 121, the map is utilizedto determine a clear path in front of the host vehicle.

FIG. 11 shows an exemplary current image (k) 500 including an exemplaryset of interest points 501 identified on the current image (k), eachinterest point 501 corresponding to a pixel. The processing module 120preferably identifies interest points 501 on each image generatedincluding the current image (k) 500 and a previous image (k−1). Aninterest point 501 is an identifiable pixel on an image and isassociated with a set of visual information, i.e., texture-richfeatures, and is associated with objects located in the field-of-view.The exemplary field-of-view or view depicted in FIG. 11 includes a roadsurface 510, sections in the road surface 515, curbs 520 and 521, lanemarkers 522, intersecting lane 526, and buildings 524 and 525. Candidateinterest points 501 are identified in the view by interest pointextraction programming from which a set of interest points 501 may beselected. The interest point extraction programming is executed on theimages by one of several known methods, e.g., a scale-invariant featuretransform (SIFT), methods employing corner detection or other shapedetection, or a Sobel filter. The interest point extraction programmingis preferably executed in the processing module 120, but may be combinedin one or more devices, e.g., implemented in software, hardware, and/orapplication-specific integrated circuitry. The interest point extractionprogramming locates candidate interest points 501 in each imagecorresponding to predetermined identifiable texture-rich features, e.g.,pixels indicating an edge, pixels indicating a transition in the visualdata, wherein potentially significant features can be identified in theview. In the exemplary view of FIG. 11, of the many identified interestpoints 501 identified, one point, 501A, is identified at a corner formedat a section 505 in curb 520.

After identifying candidate interest points 501, the interest pointextraction programming may filter the candidate interest points byremoving redundant candidate interest points, i.e., candidate interestpoints corresponding to a same feature. For example, the interest pointextraction programming may filter multiple candidate interest pointscorresponding to an edge, resulting in less candidate interest pointscorresponding to the edge. The remaining set of candidate interestpoints are the interest points for the image. In one embodiment,filtering redundant candidate interest points is executed forcomputational efficiency.

Once the interest points 501 in a current image (k) 500 are identified,the processing module 120 matches the set of interest points from thecurrent image (k) with a set of points identified from the previousimage (k−1) to determine a set of matched point pairs, corresponding tostep 109 in process 101.

A first method to determine a set of matched point pairs includes usingcorrespondence matching programming to match interest points from acurrent image (k) and a previous image (k−1). The processing module 120matches the set of interest points from the current image (k) with a setof interest points identified in the previous image (k−1) to determine aset of matched interest point pairs. FIG. 12 shows an exemplary currentimage (k) captured subsequent to the image depicted in FIG. 11. Theexemplary current image (k) 530 represents an image capturedsequentially to the image depicted in FIG. 11, wherein the perspectiveof the viewer has progressed slightly along the viewed road between theimages. In this exemplary situation, image 500 of FIG. 11, although acurrent image (k) at the time it was captured, now can act as a previousimage (k−1) to current image (k) 530. The exemplary view depicted inFIG. 12 includes road surface 510, sections in the road surface 515,curbs 520 and 521, lane markers 522, and buildings 524 and 525, asdepicted in FIG. 11, with slightly different perspectives of everyfeature in the view. In the exemplary view of FIG. 12, many identifiedinterest points 531 are identified according to methods described above,and one point, 531A, is identified at a corner formed at a section 505in curb 520. It will be noted that interest points 531 identified inimage 530 are not necessarily directly corresponding to interest points501 identified in image 500. Points 531 identified in image 530 areprocessed and filtered as described in association with points 501, asdescribed in association with FIG. 11. Preferably, several interestpoints 531 from the set of interest points from the current image (k)530 are matched with an interest point 501 from the set of interestpoints identified from the previous image (k−1) 500 to determine aplurality of matched point pairs. Each interest point comprising amatched point pair is expected to correspond to a same featureassociated with an object in the view. To determine the set of matchedpoint pairs, the set of interest points 531 from the current image (k)are compared to the set of interest points 501 identified from theprevious image (k−1) by one of several known correspondence matchingprogramming, e.g., a scale-invariant feature transform (SIFT) featurematching programming and optical flow programming. The correspondencematching programming is preferably executed in the processing module120, but may be combined in one or more devices, e.g., implemented insoftware, hardware, and/or application-specific integrated circuitry.The resulting matched point pairs correspond to a same feature locatedon both the current image (k) and the previous image (k−1) wherein thesame feature is associated with a same object in the view.

A second method to determine a set of matched point pairs includes usingtemplate matching programming to match interest points from the currentimage (k) and the previous image (k−1). In one embodiment templates aregenerated using a predetermined pixel region neighboring an interestpoint. An exemplary region 535 is depicted in association with interestpoint 531A in FIG. 12. The template matching may be determined using oneof multiple methods, including one of several known template matchingprogramming methods to find the corresponding points of interest in theprevious image, e.g., Lucas-Kanade or Horn-Schunck. Preferably, thetemplates neighboring an interest point in a current image and comparedwith templates neighboring an interest point in a previous image. Whenthe template matching programming determines that the templates match,the interest points are included in the set of matched point pairs. Theresulting matched point pairs correspond to a same feature located onboth the current image (k) and the previous image (k−1) wherein the samefeature is associated with a same object in the view.

FIGS. 13A and 13B show an exemplary matched point pair, the matchedpoint pair comprising interest point 531A from the current image (k) 530and a second interest point 501A from the previous image (k−1) 500, andan exemplary pixel region surrounding the points. According to themethod described above, pixel region 535 around interest point 531A isdepicted in FIG. 13A. As described above, interest point 531A is a pointidentified at a corner of section 505 between portions of curb 520.Pixel region 535 is selected around point 531A and preferably includesdiscernable features, textures, or patterns that can be used topositively identify the region. FIG. 13B similarly depicts interestpoint 501A and a pixel region 540 around the interest point. Because ofthe change in perspective between image 500 and image 530, it ispossible that pixel region 535 and pixel region 540 can include somedifferences, but applying methods described above, a comparison of thepixel regions and identifying features therein can be utilized as onemethod to confirm with some confidence that interest points 531A and501A represent the same feature within the view captured in images 500and 530 and may be treated as a matched pair.

After the matched point pairs are determined, the processing module 120,by applying filters removing low quality matched corresponding pointpairs indicative of poorly matched or mismatched pairs, selects highquality, preferential matched point pairs from the set of matched pointpairs, as described in exemplary step 112 above. Preferential matchedpoint pairs may be selected based upon quality control criteria. In oneembodiment, matched point pairs are analyzed and identified aspreferential matched point pairs after satisfying each criterion.

A first criterion is met when a distance between points of a matchedpoint pair is less than a threshold. The distance is determined basedupon the location of the points as if they were located on a sametwo-dimensional image coordinates. The threshold may be predeterminedand location-dependent or dynamic based upon speed of the vehicle. Asecond criterion is met when both points of a matched point pair are apredetermined threshold distance from an image boundary. Due to theperspective of the camera to objects at the edges of the image boundaryand vehicle motion, points too close to the edge either cannot locate acorresponding point, which may be located outside of the perspective ofthe camera or can create distorted estimates of object locations.Additionally, under normal vehicle operation in a forward direction, theprobability of a vehicle needing information regarding a clear pathtoward the extremes of the view is small. A third criterion is met whena difference in color between a predetermined area neighboring eachpoint of a matched point pair is less than a threshold difference incolor. Utilizing the exemplary pixel regions defined in FIG. 13, adifference in color between the pixel regions 535 and 540 can be used toincrease confidence in the matched pair. Each pixel within each of theregions can be assigned a number value based upon its correspondingcolor. The assigned number may be based upon a bit value determinedwhile capturing the image or may be based upon an indexed colorreferencing a predetermined palette. The processing module 120calculates absolute differences between the assigned number valueswithin the first area 501 and the second area 502 pixel-by-pixel, andsums the differences, which is the difference in color. The differenceis compared with the threshold difference. If the difference is lessthan the threshold difference the third criterion is met. The thresholddifference in color can be selected by any method sufficient tocalibrate accurate road presence or clear path estimation. Differentthreshold differences in color can be utilized based upon lightinglevels exterior to the vehicle, weather, speed of the vehicle, or anyother factors that affect accurate estimation of clear path presencebased upon differences in color. By judging whether the areas around theinterest points are similar, a judgment can be made regarding whetherthe same area including the feature represented by the interest pointsis being analyzed in both images.

It will be appreciated that the three criteria described above areexemplary criteria useful to judge matched pairs, but some portion ofthese criteria or additional unnamed, similar criteria can be utilizedto judge validity of matched pairs. Selection of criteria to judgematched pairs can additionally be made based upon conditions such aslight level present outside the vehicle, weather, speed of the vehicle,and any other factor affecting an ability to judge matched pairs or anurgency to quickly and accurately define a clear path.

After selecting the preferential matched point pairs, the processingmodule 120 determines positions of a plurality of points associated withfeatures of objects in the view with respect to the vehicle 100, asdescribed in exemplary step 115 above. FIG. 14 graphically illustratesuse of an exemplary current image, an exemplary previous image, andoperation of a host vehicle to determine the position of a viewedfeature. Object position in a horizontal frame of reference and a heightof the object as compared to a ground level can be determined based uponthe preferential set of matched point pairs within sequential images 317and 327 (the matched pair demonstrated in this figure through points 1and 2), a distance (d) the vehicle 100 traveled from a first position310 to a second position 320, and vehicle yaw (θ). The processing module120 executes one of several known triangulation methods to determine theposition of the point with respect to the vehicle 100 and a height ofthe point. In FIG. 14, a field-of-view 315 from 310 including an imagek−1 317 is depicted, and a field-of-view 325 from 320 including an imagek 327 is depicted. Distance d is depicted describing a distance that anobserver has traveled between points 310 and 320. Lines longitudinallyoriented to fields of view 315 and 325 are represented by lines 312 and322, respectively. An angular change in orientation or yaw (θ) isdepicted describing an angle between lines 312 and 322. Distance d canbe determined an exemplary vehicle by tracking speed of the vehiclethrough a sample time between the time at 310 and the time at 320.Similarly, θ can be determined by tracking a yaw rate of the vehiclethrough the sample time. Object being viewed 305 is depicted, and lines318 and 328 are depicted describing lines of view from points 310 and320 to object 305, respectively. Points 1 and 2 are depicted upon images317 and 327 where lines 318 and 328 intersect images 317 and 327,respectively. Distance (a) can be defined describing a location of point1 upon image 317, and distance (b) can be defined describing a locationof point 2 upon image 327. It will be appreciated that FIG. 14represents an overhead view wherein a position of object 305 can bedefined in a horizontal plane, and similar computations in a side viewof the same objects can be utilized to define a vertical position ofobject 305 with relation a known ground level for the observer. Byapplying well known methods of triangulation, positional data such asdistances a and b from sequential images and vehicular data such asvehicle speed, vehicle yaw rate, and sample time can be utilized tolocate a position of a viewed object in relation to the vehicle anddetermine relative motion of the object in relation to the vehicle.These methods of triangulation can yield a position of the object in ahorizontal plane and a height of the object in relation to a groundlevel.

Once the position and height are determined, the processing module 120may plot the points on an overhead map, as described in exemplary step118 above. FIG. 15 graphically illustrates an exemplary overhead mapdescribing vertical positions of interest points at x and y coordinatesin front of the host vehicle. Position <0, 0> on the x axis and y axiscorresponds to the second position 320 of the vehicle 100 describedhereinabove, or the current position of vehicle 100. Preferably, objectheight is classified into predetermined categories. For example, objectswith minimal height, e.g., below a predetermined threshold, may beclassified as ground, objects exceeding ground height but less than asecond predetermined threshold may be classified into a second categorywhich is close to and preferably less than a vehicle height, and objectsgreater than the second predetermined threshold may be classified into athird category. As FIG. 15 shows, objects with minimal height areclassified as ground (Ground), objects exceeding ground height but lessthan a 2 meter threshold are classified into the second category (Lessthan 2 m), and objects greater than the 2 meter threshold are classifiedinto the third category (Greater than 2 m).

After plotting the objects on the map, the processing module 120 detectsa clear path based upon features plotted in the map, as described inexemplary step 121 above. The term “clear path” as used herein is to begiven its ordinary and customary meaning to a person of ordinary skillin the art (and it is not to be limited to a special or customizedmeaning), and refers without limitation to a path free of objectsexceeding a threshold, e.g., a path free of objects classified in theabovementioned second and third categories. Features of objects that areclassified in predetermined categories, e.g., the second and thirdcategory described hereinabove are identified by the processing module120 as not-clear areas, i.e., not desirable for vehicle travel.Preferably, a predetermined area neighboring each object classified inthe predetermined categories is identified by the processing module 120as not-clear area. The processing module 120 may determine that a clearpath is present on the map using one of multiple methods. A first methodfor determining a clear path comprises comparing an anticipated pathwith the map including the not-clear areas. If the anticipated path doesnot intersect any not-clear areas then the processing module 120determines that the anticipated path is a clear path. However, if theanticipated path intersects a not-clear area then the processing module120 determines that a clear path is not present. A second method fordetermining a clear path comprises using the not-clear areas in the mapto determine a clear path. Any path not intersecting the not-clear areasmay be used as a clear path.

The above method utilizes sequential images to establish a map of objectpositions and vertical heights in front of the vehicle, such that aclear path can be defined. It will be appreciated that in any two givenimages, a particular object might not be classified as including twohigh quality interest points sufficient to be mapped in that particularanalysis. However, the above analysis occurs multiple times per secondof vehicle travel. As the vehicle travels forward through the clearpath, different perspectives upon an object will be gained and a largenumber of images will be analyzed. Travel over a path and analysis ofthe multiple iterative images through that path build a confidencethrough the analyses that no object contradicting the clear path existsin the identified clear path.

FIG. 16 shows a second processing scheme for detecting a clear path.Whereas the first processing scheme described herein utilizes texturerich methods, analyzing pixelated features that describe distinctinterest points based upon a contextual view within the image, thesecond disclosed processing scheme can be described as an exemplarytexture-less method of image analysis, filtering from an imagenon-conforming regions of the image as not belonging to a planar,consistent road surface. By filtering from an image non-conformingregions, a clear surface can be identified from the remaining image as apotential clear path for the vehicle to travel. Process 250 isillustrated in FIG. 16, and described herein as comprising discreteelements. Such illustration is for ease of description and it should berecognized that the functions performed by these elements may becombined in one or more devices, e.g., implemented in software,hardware, and/or application-specific integrated circuitry. For example,the process 250 may be executed as one or more algorithms in theprocessing module 120. The process 250 for detecting a clear pathcomprises creating a plurality of filtered images and fusing thefiltered images together to determine a clear path.

Exemplary process 250 is disclosed, including steps to employ anexemplary texture-less method of clear path detection. Process 250begins at step 253, wherein images are generated. Steps 256, 259, 262,and 265 describe alternate exemplary methods to filter an image toidentify a location of a clear path including a road surface. It will beappreciated that each of the methods processes the image in one way tohelp identify the clear path. A process could be employed with someportion of the four exemplary methods or a process could be employed toinclude unnamed but similar methods to process the image. Any method canbe employed that filters within an image a clear surface upon which aroad surface can be estimated from other portions of the image that donot indicate a potential road surface. Step 256 applies an exemplarymethod to filter an area below a horizon or vanishing point, including aroad surface that can be driven upon, from an area above the horizon orvanishing point, including sky and other vertical features that cannotbe part of a road surface. Step 259 applies a filter based upon variancein pixel intensity, based upon a premise that a road surface willinclude a large surface with a visual intensity very common across thesurface. Step 262 applies a filter based upon differencing sequentialimages, allowing analysis of changes between the images. Step 265applies a filter based upon identifying pixels representing edges ortransitions in the visual data. Applying the various methods inparallel, the results can be the fused into a single map of the image instep 268 and analyzed for visual data indicative of a clear path oftravel in step 271.

A first filtered image is created using a vanishing point, as isdescribed in exemplary step 256 above. The term “vanishing point” asused herein is a broad term, and is to be given its ordinary andcustomary meaning to one ordinarily skilled in the art, and refers to aninfinite far point on the horizon that is intersected by multipleparallel lines on the ground in the view. Identifying a road surfacecreating a clear path on which to drive is necessarily below thevanishing point or horizon line. Filtering images to only analyze anarea below the horizon line helps to clarify the pixels being analyzedto identify a road surface from irrelevant pixels. As one skilled in theart appreciates, there are many known methods for determining avanishing point and corresponding horizon line. One known methodcomprises determining the vanishing point based upon a point where thelane markers on the left and right of a host vehicle intersect eachother. The horizon line is determined based upon the vanishing point.The first filtered image consists of pixels on the current image belowthe horizon line.

The processing module 120 creates a second filtered image based uponpixel color intensity of the plurality of pixels comprising the currentimage (k), as is described in exemplary step 259 above. FIG. 17 shows anexemplary second filtered image based upon pixel color intensity. Thesecond filtered image comprises pixels selected from the plurality ofpixels. A number of methods to filter pixels are envisioned. Theprocessing module 120 compares each color intensity value of a pixelwith an associated color distribution mean e.g., a red, green, and bluecolor distribution mean. The color distribution means are an averagecolor intensity distribution for an associated color intensityassociated with pixels previously identified in clear paths. The colordistribution means may additionally be based upon historically gatheredcolor distribution means of clear path pixels of the associated colors.A pixel is selected for the second filtered image when each colorintensity value of the pixel is less than a color intensity thresholdfrom the associated color distribution mean. When one color intensityvalue of the pixel is greater than a color intensity threshold from theassociated color distribution mean, the pixel is excluded from thesecond filtered image. The color distribution mean for the associatedcolor intensity is an average color intensity value distributed in theimage. The color distribution means are initially predetermined. In oneembodiment, the processing module 120 adapts the color distribution meanfor each associated color intensity based upon the color intensitydistributions in a predetermined number of images.

As described hereinabove, each pixel comprises a plurality of colorintensity values. Preferably, the color intensity value is compared withthe associating color distribution mean. For example, for a pixelcreated using the RGB color model, a first bit set associated with a redcolor intensity is compared with the color distribution mean associatedwith the red color intensity, the bit set associated with the greencolor intensity is compared with the color distribution mean associatedwith the green color intensity, and the bit set associated with the bluecolor intensity is compared with the color distribution mean associatedwith the blue color intensity. If the difference between each colorintensity comprising the pixel and the associated color distributionmean is less than a color intensity threshold, then the pixel is usedfor the second filtered image.

Using the RGB color model, each pixel comprises a first, second, andthird color intensity value. A first color intensity is a red intensity,a second color intensity is a green intensity, and a third colorintensity is blue intensity. The pixel for the second filtered image isselected from the plurality of pixels comprising the current image (k)when the difference between a red intensity of the pixel and a red colordistribution mean is less than a red intensity threshold, and thedifference between a green intensity of the pixel and a green colordistribution mean is less than a green intensity threshold, and thedifference between a blue intensity of the pixel and a blue colordistribution mean is less than a blue intensity threshold. The followingequation expresses this process using the RGB color model:|R−R _(M)|<threshold_(R) and |G−G _(M)|<threshold_(G) and |B−B_(M)|<threshold_(B)  (4)wherein

-   -   R is a red color    -   G is a green color,    -   B is a blue color,    -   R_(M) is a color distribution mean for the red color,    -   G_(M) is a color distribution mean for the green color,    -   B_(M) is a color distribution mean for the blue color,    -   threshold_(R) is a red intensity threshold,    -   threshold_(G) is a green intensity threshold, and    -   threshold_(B) is a blue intensity threshold.

One skilled in the art will appreciate that Equation 4 above may beconfigured to be used with multiple other color models including theCMYK, YUV, and LAB color models.

The processing module 120 creates a third filtered image using thecurrent image (k) and the previous image (k−1) by differencing theimages, as is described in exemplary step 262 above. In an exemplaryprocess to create a differenced image, a third filtered image is createdas an image difference of the current image (k) and the previous image(k−1) and comprises a pixel by pixel comparison representing thedifference between the current image (k) and the previous image (k−1).Such a process is described above in association with FIGS. 6A-6C. Afirst method for determining the third filtered image comprisesdetermining a difference between individual pixels on a current image(k) and corresponding individual pixels on the previous image (k−1). Thedifference may be determined by subtracting color intensity valuesassociated with a pixel from corresponding color intensity values todetermine a color intensity difference, e.g., subtracting the red colorintensity value of a first pixel on the current image (k) from the redcolor intensity value of a first pixel on the previous image (k−1). Theresulting color intensity difference comprises the pixel value andcorresponds to the difference between the pixel on the current image (k)and the previous image (k−1). Each pixel value may be stored in an arraycorresponding to the third filtered image. In one embodiment, anabsolute value of the difference is determined and stored in the array.The resulting array comprises values representing the difference betweenthe two images and comprises the image difference. After determining thearray, the processing module 120 removes values representing pixels thatchanged less than the predetermined threshold. The resulting values can,in one embodiment, be replaced with a predetermined value, e.g., one, torepresent on a resulting image a clear map of identified differencesbetween the two utilized images.

A second exemplary method for determining the third filtered image isbased upon differencing images is described. A determination can be madeby comparing the current image (k) and a motion adjusted previous image(k−1), and differences between these images can be used to map objectsnot fitting a particular model used to generate the motion adjustedimage. One exemplary model to generate the motion adjusted imageincludes utilizing triangulation methods, such as the methods describingin association with FIG. 14, information regarding the motion of thevehicle, and an assumption that all detected points are at ground levelto predict motion of objects in the previous image (k−1). By using anassumption that all objects in the image are at ground level,predictions of motion of the objects in the resulting image areconsistent with the previous image existing as an entirely flat imageexisting at ground level. Comparison of this resulting motion adjustedprevious image (k−1) with the actual current image (k), includingperspective changes upon objects not at ground level, allows foridentification of all objects or features in the viewed images that arenot on the ground level. By subtracting pixels in the compared images,for example according to the first exemplary process for determining thethird filtered image described above, pixels representing objects not onthe ground level can be identified by the non-zero values or valuesexceeding a determinative threshold. In this way, the processing module120 can determine the third filtered image as described hereinaboveusing the current image (k) and the motion adjusted previous imageinstead of the previous image (k−1).

The processing module 120 creates a fourth filtered image based uponcolor intensity values of the plurality of pixels comprising the currentimage (k), as described in exemplary step 265 above. FIG. 18 shows afourth filtered image utilizing an edge recognition method. The fourthfiltered image comprises edges of the current image (k). To create thefourth filtered image, the processing module 120 extracts pixels fromthe image based upon color intensity values that correspond to edgesusing one of several known edge detection filters, e.g., a Sobel filter.The edge detection filter is preferably executed in the processingmodule 120, but may be combined in one or more devices, e.g.,implemented in software, hardware, and/or application-specificintegrated circuitry. In one embodiment, each pixel is analyzed using aSobel operator. The Sobel operator computes a gradient vector of colorintensity at each pixel resulting in a direction of the largest possibleincrease from light to dark and the rate of change in that direction.Points corresponding to a rate of change exceeding a threshold andcorresponding to gradient vectors at nearby pixels indicate edges andare included in the fourth filtered image. Those pixels may be replacedwith a predetermined pixel color, e.g., black.

After creating the plurality of filtered images, the processing module120 fuses the filtered images together to determine pixel locationscorresponding to a clear path, as described in exemplary step 268. Thefused filtered image comprises an image containing pixels contained inall of the plurality of filtered images, i.e., only pixels contained ata particular pixel location in each of the plurality of filtered imagesare included in the fused filtered image. Fusing the filtered imagestogether comprises determining pixel locations included on each of theplurality of filtered images. Each pixel location is compared withcorresponding pixel locations on each of the filtered images. When eachof the plurality of filtered images contains a pixel at a correspondingpixel location, that pixel is included on a fused clear path maprepresenting a clear surface on the road. The fused clear path is usedto detect a clear path for the vehicle 100. Pixels on the fused clearpath map correspond to desirable driving locations in the view.Locations on the fused clear path map without pixels correspond toundesirable driving locations in the view.

Process 250, described in FIG. 16, applies various methods to images inparallel to identify features in a view relevant to defining a clearpath. However, it will be appreciated that these methods need not beperformed in parallel, but rather the methods can be used to processimages sequentially in steps to identify features in a view relevant todefining a clear path. An exemplary process 600 for the secondprocessing scheme is depicted in FIG. 19. Process 600 initiates bygenerating a sequence of images at step 603. Each image is delivered inparallel to two steps, and the results from the two steps are fused andmade available for subsequent steps in the process. The parallel stepsinclude step 606, wherein a first filtered image according to filteringpixels by the vanishing point method described above in order to filterfrom processing the portions of the images that cannot represent a roadsurface, and step 609, wherein a second filtered image by filteringpixels according to intensity, as described above, is utilized to filterfrom processing portions of the images describing features inconsistentwith a road surface. A fused image of the first filtered image and thesecond filtered image is created in step 612, the fused image includingonly a portion of the image below the vanishing point and includingpixels with intensity consistent with a road surface. Sequential fusedimages are used, in step 615, to create a third filtered image using oneof the differencing methods described above. In step 618, the thirdfiltered image is utilized to create a fourth filtered image using theedge extraction methods described above. The results of process 600 canthen be analyzed for visual data indicative of a clear path of travel.

Processes 250 and 600 illustrate two exemplary arrangements in whichfiltering methods can be utilized in combination to process a sequenceof images to identify a clear path. However, it will be appreciated thatsuch combinations can take many forms, steps can applied in differentorders, and fewer or additional filters can be used in differentcombinations. Additionally, filters can be utilized selectively orconfigurations of processes can be utilized selectively, for example,with different combinations being utilized in the daytime and othersbeing utilized in the night. For example, some combinations can continueto analyze road surfaces illuminated by headlights whereas other methodsmight not be useful without an entirely illuminated view. In anotherexample, different combinations can be utilized when rain or snow ispresent on the road. For example, some methods and analysis methodsmight continue to be useful with a light coating of snow, for example,identifying tire paths in the snow as potential clear paths, whereasother methods might not be useful where a white covering hides mostidentifiable features. In another example, different combinations can beutilized when additional information is made available, for example,with infrared, radar, or GPS data augmenting or facilitating filteringof the images based upon the additional information. A number ofcombinations of filtering methods are envisioned, and the disclosure isnot intended to be limited to the particular embodiments describedherein.

As described above, a first processing scheme, employing texture-richmethods, and a second processing scheme, employing texture-less methods,are described, each scheme enabling identification of road featuresuseful to describe a clear path in the view of the vehicle. However, itwill be appreciated that a single analysis based upon either scheme canyield ambiguous results, for example, wherein particular lightingconditions, shadows from another vehicle, heat caused distortion of theroad surface, or other similar issues can cause portions of a roadsurface to be misidentified as not conforming to the rest of the roadsurface. One method to resolve ambiguous results is accomplished byanalyzing iterative images as the vehicle travels along the road. Itwill be appreciated that as a vehicle travels along a road, hundreds ofimages taken in quick succession can be analyzed of a particular sectionof road. As the vehicle travels forward, different lighting and exposureof the road surface as a result of different viewing perspectives willbe observed. Travel over a path and analysis of the multiple iterativeimages through that path can build confidence through the analyses thatthe particular portion of the road surface is correctly estimated as aclear path.

Another method to resolve ambiguous results of either scheme is toutilize both schemes and fuse the results, utilizing the combination ofthe two schemes to increase confidence of identification of a clearpath. Each of the schemes has advantages over the other scheme indifferent aspects of clear path detection. For example, the exemplarytexture rich method excels at detecting objects distinctly standing upfrom the ground level with rich textures. This method positivelyidentifies objects detectable at different heights and yields a map ofareas that a vehicle should not travel lest the objects collide with thevehicle. In another example, the exemplary texture-less method excels atidentifying a region wherein pixels identify a commonly appearingsurface. This method positively identifies a region wherein a roadsurface is likely to exist and maps the boundaries of this surface.

The first scheme and the second scheme can be fused in a number of ways.An image with identified points and determined heights identified withtexture-rich methods can be overlaid with a filtered image generated bytexture-less methods, and agreement of the two methods can be used todefine a clear path through the overlaid image. In an alternative methodto fuse the two schemes, the data from each of the two schemes can beused to project information upon a programmed overhead map of an area infront of the vehicle, and this overhead map including data gained fromanalysis of the two schemes can include building confidence indicationsfor regions of the map. In an alternative method to fuse the twoschemes, one scheme can be utilized as a primary or dominant scheme, andthe second scheme can be utilized or activated to analyze regions in theview identified as ambiguous or unclear. In any method to fuse the twoschemes, strengths of one processing scheme can be used to reduceweaknesses of the other processing scheme. If both schemes concur thatthe path is clear, then the processing module employing the schemes maydetermine with increased confidence that the path is desirable for thevehicle to traverse. A number of methods to fuse the identified schemesare envisioned, and the disclosure is not intended to be limited to theparticular embodiments described herein. Additionally, either scheme orboth schemes can be combined with the method employed above utilizinganalysis of patches.

An example-based method can be utilized alternatively or additionally todefine a clear path based upon an input image. An exemplaryexample-based method collects a number of sample images of views,defining a clear path for each of the sample images, matches a currentimage to one or more of the sample images, and determines a clear pathbased upon the matching. Matching the current image to one or more ofthe sample images can be accomplished, for example, by extractingfeatures from each of the sample images, extracting features from thecurrent image, comparing the extracted features from the current imageto a database of extracted features from the sample images, andselecting matching sample images to the current image. A clear path canbe selected from the best matching sample image or can be determinedbased upon a combination of the closest matches to the current image.

Hierarchical configurations of multiple clear path detection algorithmsmay be incorporated into this disclosure, including algorithms arrangedbased upon computational intensity. Less computationally intensivedetection methods can identify clear paths in the image and leaveremaining sections of the image not identified as a clear path for morecomputationally intensive analysis thereby increasing computationalefficiency for clear path identification. The particular methodsutilized and the hierarchical structure of the particular hierarchicalmethod used can differ or change. In an exemplary hierarchicalconfiguration, a patch-based clear path detection method identifiesclear paths in the image before a pixel-based clear path detectionmethod analyzes the remaining sections of the image not identified as aclear path by the patch-based method. In one embodiment, a secondexemplary hierarchical layer utilizing an example-based method furtheranalyzes sections not identified as a clear path by the pixel-basedmethod. However, a number of different hierarchical configurations areenvisioned, and the disclosure is not intended to be limited to theparticular embodiments described herein.

One skilled in the art will recognize that lighting normalization may beapplied to the captured image data. Normalization is a process whichchanges the range of pixel intensity values. The purpose of thenormalization is to bring the image into a range that is more suitablefor machine process to improve the reliability. For example, each pixelvalue is normalized to be a zero mean and unit variance to enhance theimage contrast, specifically in a low lighting environment or whencontrast is poor due to glare.

The filters and algorithms described hereinabove may take many forms.Filtering algorithms utilized to extract features often search theavailable visual information for characteristic patterns in the data,defining features by line orientation, line location, color, cornercharacteristics, other visual attributes, and learned attributes.Attributes may be developed experimentally, empirically, predictively,through modeling or other techniques adequate to accurately traindistinguishing attributes. Learned attributes may be learned by machinelearning algorithms or fuzzy logic within the vehicle adapting overtime. Additionally, learned attributes or learned landmarks can begathered from repetitive travel of a vehicle over a route and utilizedin clear path identification.

Embodiments are described above for detecting a clear path of travel fora vehicle utilizing analysis of a plurality of images generated by thecamera device located upon the vehicle. However, exemplary embodimentsdiscussed herein include a method of determining the content of an imageof a traffic infrastructure indicator to enhance the clear pathdetermination. It should be appreciated that an image of a trafficinfrastructure indication can be analyzed by the processing module todetermine the content of the traffic infrastructure indication. Thecontent of the traffic infrastructure indication can thereby beaugmented with the clear path determination to modify or enhance theclear path determination, wherein the enhanced clear path is utilized innavigation of the vehicle. For example, the clear path determination canbe inferred from the content of a traffic signal such as a stop lightindicating an approaching intersection. Likewise, a clear pathdetermination indicating an approaching traffic intersection may be usedto infer an approaching stop light. In one embodiment, area in anintersection past a stop sign can be held to be not clear until thevehicle completes the stop for the stop sign.

As discussed above, the plurality of images generated by the cameradevice are monitored and analyzed by the processing module. Based uponthe analyzed images from the camera system, a clear path can bedetermined upon which a potential road surface can be estimated fromother portions of the image that do not indicate a potential roadsurface. For example, referring to FIG. 9, the first scheme utilizestexture-rich image analysis of a field-of-view in front of the vehicle,wherein analyzed pixilated features that describe distinct interestpoints based upon a contextual view within the image are mapped andutilized to determine a clear path in front of the host vehicle. Inanother example, referring to FIGS. 16 and 19, the second schemeutilizes texture-less image analysis to filter image non-conformingregions of the image as not belonging to a planar, consistent roadsurface, wherein a plurality of filtered images are fused together todetermine a clear path. It is appreciated that the first scheme and thesecond scheme can be fused in a number of ways. For instance, an imagewith identified points and determined heights identified withtexture-rich methods can be overlaid with a filtered image generated bytexture-less methods, and agreement of the two methods can be used todefine a clear path through the overlaid image.

In an exemplary embodiment of the present disclosure, a clear path oftravel determined by analyzing images of the camera device can bemodified based upon content of an image of a traffic infrastructureindication. As discussed above, the camera system can be utilized as aninput to the processing module for determining the content of a trafficinfrastructure indication and modifying a clear path determination basedupon the content of the traffic infrastructure indication. The camerasystem includes a camera or image capturing device taking periodic orsequential images representing a view from the vehicle. Methods foranalyzing visual information are known in the art to include patternrecognition, corner detection, vertical edge detection, vertical objectrecognition, and other methods. Additionally, image recognitionfrequently includes programming to look for changes in contrast or colorin an image indicating vertical lines, edges, corners or other patternsindicative of traffic infrastructure indications. Such image recognitionmethods can include methods to discern textual messages from the signs.In addition, particular sign types, logos, or trademarks can be storedin a database for comparison and cross referenced for context orsignificance. However, it will be appreciated that high-resolutionvisual representations of the field in front of a vehicle refreshing ata high rate necessary to appreciate motion in real-time include a verylarge amount of information to be analyzed. Real-time analysis of visualinformation can be prohibitive. Methods of fusing input from a camerasystem created by methods such as track fusion are known in the art andwill not be discussed in detail herein. Utilizing fusing input helps todiscern moving objects such as vehicles from stationary trafficinfrastructure indications such as roadside sign.

Camera data can be utilized to detect traffic infrastructure indicatorssuch as roadside signs and traffic lights. It will be appreciated thatother devices can be utilized to detect traffic infrastructureindicators. For example, LIDAR can be utilized to recognize the presenceof signs and some context from the signs, for example, discerning theshape of a stop sign. Vehicle to infrastructure (V2I) communications canbe utilized to communicate a content of a street sign or a traffic lightto passing vehicles, and other onboard devices such as a radar systemcan be utilized to localize the traffic commands, for example to anintersection or a sign post location. Vehicle to vehicle (V2V)communications can be utilized to communicate data between vehicles, forexample, with one vehicle utilizing a camera system communicatinginformation about an intersection to following vehicles approaching theintersection. In another example, vehicles stopped at a stop sign at anintersection can utilize V2V communications to negotiate an order forthe vehicles to proceed through their stop signs, with the clear path ofeach vehicle reflecting the negotiation. In one embodiment, anintersection can be determined to be not a clear path until thenegotiation is made for the vehicle to proceed. Similarly, a red lightcan be utilized to classify the intersection as not clear until thelight turns green. A number of devices and additional embodimentsutilizing the devices to detect traffic infrastructure indications areenvisioned, and the disclosure is not intended to be limited to theexemplary embodiments described herein.

Traffic infrastructure indications include numerous possibilities.Traffic infrastructure indications can include, inter alia, roadsidesigns and traffic signals. Roadside signs can include speed limitindications or road classification indicators, such as interstatehighway indicators. Such indicators can be utilized with a clear pathanalysis to forecast a type of road being travelled upon. For example, aroadway with a high speed limit or with a highway designation isunlikely to include a sharp turn in excess of certain guidelines.Roadside signs further include approaching road geometries relating totraffic lanes such as mandatory lane curvatures, turn only lanes, oneway, keep right, keep left, and exit/enter roadside signs. Roadsidesigns can further include warnings of approaching road geometries suchas added left/right lane, left/right lane ending, no outlet, side roadleft/right, railroad crossings, reverse curves, reverse turn left/right,lane ending and approaching intersections. Clear path analysis caninclude enhanced traffic infrastructure indication detection, focusingcomputational resources based upon forecasted road geometry. Forexample, a patch-based grid pattern can be selected to include patchesparticularly arrayed to a left side of a field-of-view if an upcomingleft turn in the road is forecasted. A texture-rich analysis, in asimilar situation, detecting lane markings and accompanying indicatorsfollowing the left turn, can include dedicated analysis leading the lanemarkers around the turn. By focusing computational resources based upona predetermined road geometry, the overall detection can be made moreefficient. Additionally, it will be appreciated that clear pathdetection involves developing different hypotheses of the structure ofthe detected roadway and selecting between these different hypotheses todetermine the clear path. Additional information including informationthat is available from traffic infrastructure devices can give more orless weight to different hypotheses. It should be appreciated that a 3-Dmapping device and a GPS device can additionally or alternatively beutilized to determine the content of the roadside signs detected by thecamera system, where the content of the roadside signs is stored in the3-D mapping device and/or the GPS device. For example, an approachingroundabout could be used to guide clear path analysis towards the rightbased on a priori known structure of a roundabout. Additionally, contentof roadside signs relating to road construction geometries can also beutilized to modify the determined clear path. For example, a speed limitmay be reduced in a construction zone due to the presence of roadworkers. Although a clear path can be determined by the camera systemutilizing one of the schemes above, the content of a roadside signindicating a reduced speed in the construction zone can be augmented formutual benefit refinement with the determined clear path and utilized toassist in navigation of the vehicle. In another example, constructionzone signs frequently alert to shifting lanes, lane ending, instructionsto maintain a current lane, and other similar indications. Suchindications can be utilized to enhance clear path detection with thesign instructions. Similarly, the detected clear path can be utilized toaugment comprehension of the indicator instructions, for example, withthe boundaries of the detected clear path being likely to includeconstruction zone indicators.

Referring to FIG. 20, an exemplary embodiment illustrating a processscheme 700 for determining an enhanced clear path based upon mutualbenefit refinement between roadside sign analysis and map fusion of adetected clear path. Exemplary process 700 begins at step 702, whereinimages are generated. It will be appreciated at step 706, that a clearpath can be detected by the first processing scheme, employingtexture-rich methods, or the second processing scheme, employingtexture-less methods, wherein each scheme enables identification of roadfeatures useful to describe a clear path in the view of the vehicle.Clear path analysis by the fusion of texture-rich and texturelessmethods is given as an exemplary analysis, but it will be appreciatedthat any of the above described analyses can be used. At step 710, thefirst scheme and the second scheme can be fused in a number of ways.These methods are discussed above, and will not be discussed in detailherein. At step 704, a roadside sign can be detected and analyzed atstep 708. As discussed above, a detected roadside sign can be analyzedto determine the content of the roadside sign. The content of theroadside sign can be determined utilizing pattern recognition, cornerdetection, vertical edge detection, vertical object recognition, imagerecognition and/or image recognition methods including methods todiscern textual messages from the signs. For instance, image recognitioncould be utilized to discern a stop sign. Additionally, the roadsidesign can be analyzed to determine the content by storing the image ofthe roadside sign in a database for comparison and cross referenced forcontext or significance. At step 712, the content of the analyzedroadside sign can be augmented or fused with the detected and fusedclear path for mutual benefit refinement. Mutual benefit refinementprovides for confidence to the detected clear path, and additionally,can be utilized to infer the possibility of roadside sign locationsalong the roadway. At step 714, an enhanced clear path is determinedbased upon the mutual benefit refinement determined at step 712.

Traffic infrastructure indications can also include traffic signalindications. Traffic signal indications can include, but are not limitedto, traffic lights, railroad crossing lights, stop lights and cautionaryintersection lights. Referring to FIG. 21, an exemplary embodimentillustrating a process scheme 800 for determining an enhanced clear pathbased upon mutual benefit refinement between traffic signal stateanalysis and map fusion of a detected clear path. Exemplary process 800begins at step 802, wherein images are generated. It will be appreciatedat step 806, that a clear path can be detected by fusion of an exemplarypatch-based method and an exemplary pixel-based method. At step 810, thetwo detection methods can be fused in a number of ways. These methodsare discussed above, and will not be discussed in detail herein. At step804, a traffic signal, for instance a traffic light, can be detected andanalyzed at step 808. At step 814, the traffic signal can be detectedutilizing pattern recognition, corner detection, vertical edgedetection, vertical object recognition, image recognition, imagerecognition and/or storing the image of the traffic signal in a databasefor comparison and cross referenced for context or significance. Asdiscussed above, a detected traffic signal can be analyzed to determinethe content of the traffic signal. For instance, analysis of a trafficlight can discern a proceed signal, stop signal or caution signal byutilizing color contrast methods or measuring light intensity todetermine a green, red or amber color signal. At step 812, the contentof the analyzed traffic signal can be augmented or fused with thedetected and fused clear path for mutual benefit refinement. Mutualbenefit refinement provides for confidence to the detected clear path,and additionally, can be utilized to infer the possibility of trafficsignal locations on the roadway. For example, although a clear path maybe determined, a red traffic signal indication can be augmented with thedetected clear path to inform the vehicle that traffic may beintersecting the clear path. At step 814, an enhanced clear path isdetermined based upon the mutual benefit refinement determined at step812.

By monitoring the images from the camera system and applying analysis tothe monitored images in order to identify an image of a trafficinfrastructure indication, and further analyzing the image of thetraffic infrastructure indication to determine the content of thetraffic infrastructure indication, the determined clear path can bemodified based upon the content of the traffic infrastructure indicationand utilized in navigation of the vehicle. The analyzed image of thetraffic infrastructure indication can describe a threshold condition.

The threshold condition may be preset, learned, and/or selectable basedupon numerous informational inputs. Threshold conditions are set tominimize non-critical or non-desirable distractions upon the vehicle,wherein the image of a traffic infrastructure indication describing athreshold condition is processed and augmented with a clear pathdetermination to determine an enhanced clear path, wherein the enhancedclear path is utilized in navigation of the vehicle. It is understoodthat the vehicle is capable of operating in an autonomous orsemi-autonomous driving mode. The vehicle includes an autonomoussteering device and a speed control device that are operative based uponwhether a clear path has been determined or not, and additionally takinginto consideration the content of traffic infrastructure indicationsdescribing threshold conditions provides increased confidence in theclear path captured by the camera device and analyzed by the processingmodule.

Clear path detection can be utilized to enhance detection andunderstanding the context of roadside signs. For example, the currentspeed of a vehicle may be monitored while the camera system captures animage of a roadside sign indicating the speed limit. The speed limitsign captured by the camera system may be analyzed by patternrecognition to discern a speed limit indication. The discerned speedlimit is thereby compared to the monitored speed of the vehicle toindicate a threshold condition based upon the comparing. However, thedetection of the speed limit can be in error. Analysis of the clear pathof travel, for example, describing an approaching sharp turn in the roador a pattern describing a special zone such as a parking lot or a schoolzone, the monitored speed limit can be checked against the informationdiscernable by the clear path analysis and invalidated as appropriate.

In another example, the camera system can capture an image of a trafficsignal indicator while the current speed of the vehicle and the distanceof the vehicle to the traffic signal are monitored. The monitored imageof the traffic signal indicator can be stored in a database forcomparison and cross referenced for context to indicate a traffic light.The speed of the vehicle and the distance of the vehicle from thetraffic light can be compared and a threshold condition can be indicatedbased upon the comparing. For instance, a threshold condition can bedetermined, based upon the speed of the vehicle and the distance thevehicle is from the traffic light, describing that the vehicle is notplanning to stop at the traffic light. The threshold condition can beaugmented with the clear path determination to generate a mutual benefitrefinement based upon the augmentation between the determined thresholdand the determined clear path.

In another example, the camera system can be utilized to capture animage of a roadside sign while informational inputs can be monitored todescribe a lane in which the vehicle is currently traveling. A laneending roadside sign, informational billboard or configurable trafficmessage sign, or electronically transmitted vehicle to infrastructurecommunication can be analyzed to indicate a threshold condition based onmonitoring informational inputs relating to the lane in which thevehicle is traveling and vacancies in neighboring lanes to merge into.For example, utilizing visual informational inputs from the camerasystem, comprehension of which lane the vehicle is currently travelingis determined and a threshold condition can be indicated when ananalyzed image of a roadside sign indicates that the lane the vehicle iscurrently travelling is ending and a merge into a neighboring lane isrequired. The threshold condition can be augmented with the clear pathdetermination to generate a mutual benefit refinement based upon theaugmentation between the determined threshold and the determined clearpath.

In another exemplary embodiment, threshold conditions based uponmonitored informational inputs indicating augmented road signs in riskof being violated can be implemented. A stop sign captured by the camerasystem can be analyzed using image recognition indicating colors andshapes indicative of stop signs. Informational inputs relating to thecurrent speed of the vehicle and stopping distance to the stop sign canbe monitored. It is appreciated that the stopping distance to the stopsign can be determined utilizing GPS information, 3D map informationand/or radar returns. A threshold condition can be indicated if it isdetermined that the vehicle is in risk of violating the stop sign basedon the monitored informational inputs relating to the current speed ofthe vehicle and the stopping distance to the stop sign. The thresholdcondition can be augmented with the clear path determination to generatea mutual benefit refinement based upon the augmentation between thedetermined threshold and the determined clear path.

Step 812 of FIG. 21 and step 712 of FIG. 20 illustrate mutual benefitsrefinement, inputting a clear path and traffic infrastructure indicationinformation, to output an enhanced clear path. As will be appreciatedfrom the above described methods, clear path can benefit from detectionof traffic infrastructure indications, augmenting an understanding ofwhat areas are likely to be a clear path. It will be appreciated thatenhanced traffic infrastructure indication comprehension can similarlybe accomplished by analysis of the identified clear path. For example,in a construction area, clear path indications showing shifting lanes oftraffic or vehicles undergoing unexpected changes in motion can beutilized to augment detection and analysis of the construction area.Computational resources can be devoted to the analysis of the area basedupon the clear path information. Such an analysis can be iterativelyreinforced with subsequent images, building confidence along the routeof travel of both the clear path and the traffic infrastructureindication. Additionally, a single image can be iteratively examined,with multiple iterations of clear path analysis and trafficinfrastructure indication analysis reinforcing overall analysis of thesingle image. Such an analysis loop can beneficially refine startingassumptions and results of the analyses. FIG. 22 graphically depictssuch an iterative analysis, in accordance with the present disclosure.Step 812 is depicted, including traffic infrastructure indicationrefinement in step 809 and clear path refinement in step 811. Step 812outputs both the enhanced clear path, as described above, andadditionally exemplary enhanced object tracking information.

FIG. 23 depicts an exemplary flowchart describing the above definedmethods shown in FIG. 20 and FIG. 21 for determining an enhanced clearpath of travel. Step 902 generates images by a camera device asdescribed above. Step 904 monitors the generated images from the cameradevice. Step 906 analyzes the images comprising determining a clear pathupon which a potential road surface can be estimated from other portionsof the images that do not indicate a potential road surface. Step 908monitors a device generating data describing a traffic infrastructureindication and step 910 analyzes the data describing the trafficinfrastructure indication to describe a threshold condition. Step 912analyzes the threshold condition and the clear path to determine anenhanced clear path and step 914 utilizes the enhanced clear path innavigation of the vehicle.

The disclosure has described certain preferred embodiments andmodifications thereto. Further modifications and alterations may occurto others upon reading and understanding the specification. Therefore,it is intended that the disclosure not be limited to the particularembodiment(s) disclosed as the best mode contemplated for carrying outthis disclosure, but that the disclosure will include all embodimentsfalling within the scope of the appended claims.

The invention claimed is:
 1. Method for determining an enhanced clearpath of travel for a vehicle utilizing analysis of a plurality of imagesgenerated by a camera device located upon the vehicle, the methodcomprising: within a processing module: monitoring images from thecamera device; analyzing the images comprising determining a clear pathupon which a potential road surface can be estimated from other portionsof the images that do not indicate a potential road surface; determininga traffic infrastructure indication, comprising; monitoring a devicegenerating data describing a traffic infrastructure indication;comparing the traffic infrastructure indication with the determinedclear path; determining the determined clear path describes a conditionthat invalidates the traffic infrastructure indication; enhancing thetraffic infrastructure indication by modifying it to correspond with thedetermined clear path; analyzing the data describing the enhancedtraffic infrastructure indication to describe a threshold condition,comprising; determining a current vehicle operating parameter; comparingthe determined current vehicle operating parameter to the enhancedtraffic infrastructure indication; and indicating the thresholdcondition based upon the comparing of the determined current vehicleoperating parameter to the enhanced traffic infrastructure indication,said threshold condition selected to minimize at least one ofnon-critical and non-desirable distractions upon the vehicle; andanalyzing the threshold condition and the clear path to determine anenhanced clear path, comprising augmenting the threshold condition withthe clear path to generate a mutual benefit refinement based upon theaugmenting of the threshold condition with the clear path, by; comparingthe clear path to the indicated threshold condition; and enhancing theclear path by modifying the clear path based upon the comparing; andutilizing the enhanced clear path in navigation of the vehicle,including controlling vehicle control mechanisms based upon the enhancedclear path and operating the vehicle to travel along the enhanced clearpath of travel.
 2. The method of claim 1, further comprising: monitoringinformational inputs describing operation of the vehicle; and whereinanalyzing the data describing the enhanced traffic infrastructureindication to describe the threshold condition is based upon theinformational inputs.
 3. The method of claim 2, wherein monitoringinformational inputs describing operation of the vehicle comprisesmonitoring a speed of the vehicle and a distance of the vehicle from thetraffic infrastructure indication; and wherein analyzing the datadescribing the enhanced traffic infrastructure indication to indicatethe threshold condition comprises analyzing the data describing theenhanced traffic infrastructure indication utilizing a database todiscern a traffic signal, comparing the monitored speed of the vehicleand the distance of the vehicle from the traffic signal, and indicatingthe threshold condition based upon the comparing.
 4. The method of claim1, wherein analyzing the data describing the enhanced trafficinfrastructure indication to describe the threshold condition comprisesdetermining a current speed limit; and wherein analyzing the thresholdcondition and the clear path to determine the enhanced clear pathcomprises: comparing the clear path to the current speed limit; andmodifying the clear path based upon the comparing.
 5. The method ofclaim 1, wherein analyzing the data describing the enhanced trafficinfrastructure indication to describe the threshold condition comprisesdetermining an upcoming road geometry; and wherein analyzing thethreshold condition and the clear path to determine the enhanced clearpath comprises: comparing the clear path to the upcoming road geometry;and modifying the clear path based upon the comparing.